{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "🛠 08. Introduction to NLP (Natural Language Processing) in TensorFlow Exercise Solution.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ashikshafi08/Learning_Tensorflow/blob/main/Exercise%20Solutions/%F0%9F%9B%A0_08_Introduction_to_NLP_(Natural_Language_Processing)_in_TensorFlow_Exercise_Solution.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nHwk8Cgs6vnj"
      },
      "source": [
        "# 🛠 08. Introduction to NLP (Natural Language Processing) in TensorFlow Exercises\n",
        "\n",
        "1. Rebuild, compile and train `model_1`, `model_2` and `model_5` using the Keras Sequential API instead of the Functional API.\n",
        "2. Retrain the baseline model with 10% of the training data. How does perform compared to the Universal Sentence Encoder model with 10% of the training data?\n",
        "\n",
        "3. Try fine-tuning the TF Hub Universal Sentence Encoder model by setting training=True when instantiating it as a Keras layer\n",
        "\n",
        "```\n",
        "# We can use this encoding layer in place of our text_vectorizer and embedding layer\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=True) # turn training on to fine-tune the TensorFlow Hub model\n",
        "\n",
        "```\n",
        "\n",
        "4. Retrain the best model you've got so far on the whole training set (no validation split). Then use this trained model to make predictions on the test dataset and format the predictions into the same format as the `sample_submission.csv` file from Kaggle (see the Files tab in Colab for what the `sample_submission.csv` file looks like). Once you've done this, [make a submission to the Kaggle competition](https://www.kaggle.com/c/nlp-getting-started/data), how did your model perform?\n",
        "\n",
        "5. Combine the ensemble predictions using the majority vote (mode), how does this perform compare to averaging the prediction probabilities of each model?\n",
        "\n",
        "6. Make a confusion matrix with the best performing model's predictions on the validation set and the validation ground truth labels."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7GRfIvxC610_"
      },
      "source": [
        "## Getting the data "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zvTvHV8LBroJ"
      },
      "source": [
        "# Importing the needed packages \n",
        "import tensorflow as tf\n",
        "import pandas as pd \n",
        "import matplotlib.pyplot as plt "
      ],
      "execution_count": 1,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QcOZFdVgBbBW",
        "outputId": "04abfd85-e00b-4b99-d488-1839601ce293"
      },
      "source": [
        "# Downloading the helper function file \n",
        "!wget https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
        "\n",
        "# Importing series of function for our use \n",
        "from helper_functions import unzip_data, plot_loss_curves, compare_historys\n",
        "\n",
        "# Download data (same as from Kaggle)\n",
        "!wget \"https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\"\n",
        "\n",
        "# Unzip data\n",
        "unzip_data(\"nlp_getting_started.zip\")\n"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-07-16 09:59:03--  https://raw.githubusercontent.com/mrdbourke/tensorflow-deep-learning/main/extras/helper_functions.py\n",
            "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.111.133, 185.199.109.133, ...\n",
            "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 10246 (10K) [text/plain]\n",
            "Saving to: ‘helper_functions.py’\n",
            "\n",
            "\rhelper_functions.py   0%[                    ]       0  --.-KB/s               \rhelper_functions.py 100%[===================>]  10.01K  --.-KB/s    in 0s      \n",
            "\n",
            "2021-07-16 09:59:03 (108 MB/s) - ‘helper_functions.py’ saved [10246/10246]\n",
            "\n",
            "--2021-07-16 09:59:04--  https://storage.googleapis.com/ztm_tf_course/nlp_getting_started.zip\n",
            "Resolving storage.googleapis.com (storage.googleapis.com)... 172.217.218.128, 142.250.153.128, 173.194.69.128, ...\n",
            "Connecting to storage.googleapis.com (storage.googleapis.com)|172.217.218.128|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 607343 (593K) [application/zip]\n",
            "Saving to: ‘nlp_getting_started.zip’\n",
            "\n",
            "nlp_getting_started 100%[===================>] 593.11K  --.-KB/s    in 0.007s  \n",
            "\n",
            "2021-07-16 09:59:04 (86.5 MB/s) - ‘nlp_getting_started.zip’ saved [607343/607343]\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hmnaZX2XBy8e",
        "outputId": "3795c103-48e8-4a44-ddad-cdabf150f9da"
      },
      "source": [
        "# Loading in the data \n",
        "train_df = pd.read_csv('train.csv')\n",
        "test_df = pd.read_csv('test.csv')\n",
        "\n",
        "# Shuffling the training dataframe \n",
        "train_df_shuffled = train_df.sample(frac =1 , random_state= 42)\n",
        "\n",
        "# Split our data into training and test sets \n",
        "from sklearn.model_selection import  train_test_split\n",
        "train_sentences , val_sentences , train_labels , val_labels = train_test_split(train_df_shuffled['text'].to_numpy() , \n",
        "                                                                               train_df_shuffled['target'].to_numpy(), \n",
        "                                                                               test_size = 0.1 , \n",
        "                                                                               random_state = 42)\n",
        "\n",
        "# Checking the shapes \n",
        "train_sentences.shape , val_sentences.shape , train_labels.shape , val_labels.shape"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "((6851,), (762,), (6851,), (762,))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-J5Z-1JCdHp"
      },
      "source": [
        "Let's convert our text into numbers by\"\n",
        "- Text Vectorization --> Turns our text into tokens\n",
        "- Embedding --> Turns out tokens into a vector "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0MvOoeVrD_Jq"
      },
      "source": [
        "# Setting up text vectorization \n",
        "from tensorflow.keras.layers.experimental.preprocessing import TextVectorization\n",
        "max_vocab_length = 10000 \n",
        "max_length = 15 \n",
        "\n",
        "text_vectorizer = TextVectorization(max_tokens = max_vocab_length , \n",
        "                                    output_mode = 'int', \n",
        "                                    output_sequence_length = max_length)\n",
        "\n",
        "# Fit the text vectorizer to the training text\n",
        "text_vectorizer.adapt(train_sentences)\n",
        "\n",
        "# Creating a embedding layer \n",
        "from tensorflow.keras import layers \n",
        "\n",
        "embedding = layers.Embedding(input_dim= max_vocab_length , \n",
        "                             output_dim = 128 , \n",
        "                             embeddings_initializer = 'uniform', \n",
        "                             input_length = max_vocab_length)\n"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LLiaRVuqEkBX"
      },
      "source": [
        "Alright now it's time to jump into exercises! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cqcNtOX8FHmk"
      },
      "source": [
        "### 1. Rebuild, compile and train `model_1`, `model_2` and `model_5` using the Keras Sequential API instead of the Functional API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W432ozz2mgLn"
      },
      "source": [
        "**Model_1**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "VQxywED3FJX0",
        "outputId": "5c34a5a7-ef89-42fe-ff74-0685902e782e"
      },
      "source": [
        "# Building the model 1 (A simple Dense model)\n",
        "model_1 = tf.keras.Sequential([\n",
        "    layers.Input(shape = (1, ) , dtype = 'string'), \n",
        "    text_vectorizer , \n",
        "    embedding , \n",
        "    layers.GlobalAveragePooling1D() , \n",
        "    layers.Dense(1, activation = 'sigmoid')\n",
        "    ])\n",
        "    \n",
        "# Compiling the model \n",
        "model_1.compile(loss = tf.keras.losses.BinaryCrossentropy() , \n",
        "                optimizer = tf.keras.optimizers.Adam() , \n",
        "                metrics = ['accuracy'])\n",
        "\n",
        "# Fitting the model \n",
        "model_1_history =  model_1.fit(train_sentences, \n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels))"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 3s 9ms/step - loss: 0.6129 - accuracy: 0.6888 - val_loss: 0.5380 - val_accuracy: 0.7585\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.4431 - accuracy: 0.8178 - val_loss: 0.4697 - val_accuracy: 0.7874\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.3475 - accuracy: 0.8596 - val_loss: 0.4590 - val_accuracy: 0.7861\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.2847 - accuracy: 0.8880 - val_loss: 0.4709 - val_accuracy: 0.7927\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.2383 - accuracy: 0.9101 - val_loss: 0.4783 - val_accuracy: 0.7795\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkMPmldXkqIf"
      },
      "source": [
        "**Model_2**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qeko8mC4l7ZY",
        "outputId": "45bf1c33-87e0-4534-d8ed-f1cf4ae1e6bc"
      },
      "source": [
        "# Building the model 2 (An LSTM model)\n",
        "model_2 = tf.keras.Sequential([\n",
        "    layers.Input(shape = (1, ) , dtype = 'string'), \n",
        "    text_vectorizer , \n",
        "    embedding , \n",
        "    layers.LSTM(64), \n",
        "    #layers.GlobalAveragePooling1D() , \n",
        "    layers.Dense(1, activation = 'sigmoid')\n",
        "    ])\n",
        "    \n",
        "# Compiling the model \n",
        "model_2.compile(loss = tf.keras.losses.BinaryCrossentropy() , \n",
        "                optimizer = tf.keras.optimizers.Adam() , \n",
        "                metrics = ['accuracy'])\n",
        "\n",
        "# Fitting the model \n",
        "model_2_history =  model_1.fit(train_sentences, \n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.2016 - accuracy: 0.9294 - val_loss: 0.5041 - val_accuracy: 0.7861\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.1736 - accuracy: 0.9402 - val_loss: 0.5268 - val_accuracy: 0.7861\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.1495 - accuracy: 0.9480 - val_loss: 0.5566 - val_accuracy: 0.7795\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.1315 - accuracy: 0.9550 - val_loss: 0.5868 - val_accuracy: 0.7756\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.1171 - accuracy: 0.9618 - val_loss: 0.6140 - val_accuracy: 0.7743\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uc6m3A05m4XP"
      },
      "source": [
        "**Model_5** "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DdYJMmm8nFIG",
        "outputId": "83daf8ea-171b-4aee-98d8-f58c3dfc15a5"
      },
      "source": [
        "# Building the model 5 (Conv1D Model)\n",
        "# Building the model 1 (A simple Dense model)\n",
        "model_5 = tf.keras.Sequential([\n",
        "    layers.Input(shape = (1, ) , dtype = 'string'), \n",
        "    text_vectorizer , \n",
        "    embedding , \n",
        "    layers.Conv1D(32 , 5 , activation= 'relu'),\n",
        "    layers.GlobalMaxPool1D() , \n",
        "    layers.Dense(1, activation = 'sigmoid')\n",
        "    ])\n",
        "    \n",
        "# Compiling the model \n",
        "model_5.compile(loss = tf.keras.losses.BinaryCrossentropy() , \n",
        "                optimizer = tf.keras.optimizers.Adam() , \n",
        "                metrics = ['accuracy'])\n",
        "\n",
        "# Fitting the model \n",
        "model_5_history =  model_1.fit(train_sentences, \n",
        "                              train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels))"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:Please add `keras.layers.InputLayer` instead of `keras.Input` to Sequential model. `keras.Input` is intended to be used by Functional model.\n",
            "Epoch 1/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.1049 - accuracy: 0.9644 - val_loss: 0.6500 - val_accuracy: 0.7717\n",
            "Epoch 2/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0956 - accuracy: 0.9689 - val_loss: 0.6794 - val_accuracy: 0.7703\n",
            "Epoch 3/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0867 - accuracy: 0.9710 - val_loss: 0.7105 - val_accuracy: 0.7690\n",
            "Epoch 4/5\n",
            "215/215 [==============================] - 2s 8ms/step - loss: 0.0805 - accuracy: 0.9726 - val_loss: 0.7373 - val_accuracy: 0.7664\n",
            "Epoch 5/5\n",
            "215/215 [==============================] - 2s 9ms/step - loss: 0.0741 - accuracy: 0.9753 - val_loss: 0.7741 - val_accuracy: 0.7664\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y9213rMcna1E"
      },
      "source": [
        "### 2. Retrain the baseline model with 10% of the training data. How does perform compared to the Universal Sentence Encoder model with 10% of the training data?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DKDgH5wfoLmz",
        "outputId": "c84a8abe-7805-423e-8055-456d89c9f053"
      },
      "source": [
        "# Making a better dataset split (no data leakage)\n",
        "train_10_percent_split = int(0.1 * len(train_sentences)) # Directly taking 10% from our train data\n",
        "print(train_10_percent_split)\n",
        "\n",
        "# Splitting our train data with the actual train_data (no double dipping this time)\n",
        "train_sentences_10_percent = train_sentences[:train_10_percent_split]\n",
        "\n",
        "# Doing the same but with the train labels \n",
        "train_labels_10_percent = train_labels[:train_10_percent_split]\n",
        "len(train_labels_10_percent) , train_labels_10_percent[:5]"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "685\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(685, array([0, 0, 1, 0, 0]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eB2Jbhkfoa33",
        "outputId": "c8ad7b66-8450-439e-85fe-08636a6af992"
      },
      "source": [
        "# Building a base line model \n",
        "from sklearn.feature_extraction.text import TfidfVectorizer # (turn text into numbers)\n",
        "from sklearn.naive_bayes import MultinomialNB\n",
        "from sklearn.pipeline import Pipeline \n",
        "\n",
        "# Create tokenization and modelling pipeline\n",
        "baseline_model = Pipeline([\n",
        "                  ('tfidf' , TfidfVectorizer()) , # Convert words to numbers using tfidf\n",
        "                  ('clf' , MultinomialNB()), # Model the text \n",
        "                ])\n",
        "\n",
        "# Fit the pipeline to the training data \n",
        "baseline_model.fit(train_sentences_10_percent , train_labels_10_percent)\n",
        "\n",
        "# Evaluating our baseline model \n",
        "baseline_score = baseline_model.score(val_sentences , val_labels)\n",
        "baseline_score"
      ],
      "execution_count": 9,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0.7020997375328084"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "90u-fUWso-E3"
      },
      "source": [
        "Our **Universal Sentence Encoder** model outperformed our **baseline model** with just 10% of the training data. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ClyFKWq-pXt2"
      },
      "source": [
        "### 3. Try fine-tuning the TF Hub Universal Sentence Encoder model by setting training=True when instantiating it as a Keras layer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pkXu2e35DKBd"
      },
      "source": [
        "# Turn training on to fine-tune the TensorFlow Hub model\n",
        "import tensorflow_hub as hub\n",
        "sentence_encoder_layer = hub.KerasLayer(\"https://tfhub.dev/google/universal-sentence-encoder/4\",\n",
        "                                        input_shape=[],\n",
        "                                        dtype=tf.string,\n",
        "                                        trainable=True) "
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Kk3qw0v_D8UI",
        "outputId": "e504971e-03df-4514-87a5-018b8a1ef341"
      },
      "source": [
        "# Create model using the Sequential API\n",
        "use_model = tf.keras.Sequential([\n",
        "  sentence_encoder_layer, \n",
        "  layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "\n",
        "# Compile model\n",
        "use_model.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Train a classifier on top of pretrained embeddings\n",
        "use_model_history = use_model.fit(train_sentences_10_percent,\n",
        "                              train_labels_10_percent,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels))"
      ],
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "22/22 [==============================] - 33s 1s/step - loss: 0.6195 - accuracy: 0.6599 - val_loss: 0.5592 - val_accuracy: 0.7717\n",
            "Epoch 2/5\n",
            "22/22 [==============================] - 27s 1s/step - loss: 0.4614 - accuracy: 0.8277 - val_loss: 0.4988 - val_accuracy: 0.7861\n",
            "Epoch 3/5\n",
            "22/22 [==============================] - 27s 1s/step - loss: 0.3211 - accuracy: 0.8964 - val_loss: 0.4728 - val_accuracy: 0.7979\n",
            "Epoch 4/5\n",
            "22/22 [==============================] - 27s 1s/step - loss: 0.1868 - accuracy: 0.9620 - val_loss: 0.5172 - val_accuracy: 0.7743\n",
            "Epoch 5/5\n",
            "22/22 [==============================] - 27s 1s/step - loss: 0.1163 - accuracy: 0.9854 - val_loss: 0.5333 - val_accuracy: 0.7822\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XAdWe94gFV-6"
      },
      "source": [
        "### 4. Retrain the best model you've got so far on the whole training set (no validation split). \n",
        "\n",
        "Then use this trained model to make predictions on the test dataset and format the predictions into the same format as the sample_submission.csv file from Kaggle (see the Files tab in Colab for what the sample_submission.csv file looks like). Once you've done this, make a submission to the Kaggle competition, how did your model perform?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "E_9KU0nxEf8Z",
        "outputId": "6bb00ad9-2ade-432e-9567-ad2e02fcae10"
      },
      "source": [
        "whole_train_sentences = train_df_shuffled['text'].to_numpy()\n",
        "whole_train_labels =  train_df_shuffled['target'].to_numpy() \n",
        "\n",
        "len(whole_train_sentences) , len(whole_train_labels)"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(7613, 7613)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "IkQU4qohFqZZ",
        "outputId": "010407db-7807-4380-a1dc-3cc279840dbc"
      },
      "source": [
        "# Create model using the Sequential API\n",
        "model = tf.keras.Sequential([\n",
        "  sentence_encoder_layer, \n",
        "  layers.Dense(64 , activation ='relu'),\n",
        "  layers.Dense(1, activation=\"sigmoid\")\n",
        "])\n",
        "\n",
        "# Compile model\n",
        "model.compile(loss=\"binary_crossentropy\",\n",
        "                optimizer=tf.keras.optimizers.Adam(),\n",
        "                metrics=[\"accuracy\"])\n",
        "\n",
        "# Train a classifier on top of pretrained embeddings\n",
        "model_history =model.fit(whole_train_sentences,\n",
        "                              whole_train_labels,\n",
        "                              epochs=5,\n",
        "                              validation_data=(val_sentences, val_labels))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/5\n",
            "238/238 [==============================] - 296s 1s/step - loss: 0.4133 - accuracy: 0.8235 - val_loss: 0.2616 - val_accuracy: 0.9016\n",
            "Epoch 2/5\n",
            "238/238 [==============================] - 302s 1s/step - loss: 0.1956 - accuracy: 0.9274 - val_loss: 0.0638 - val_accuracy: 0.9843\n",
            "Epoch 3/5\n",
            "238/238 [==============================] - 299s 1s/step - loss: 0.0675 - accuracy: 0.9777 - val_loss: 0.0275 - val_accuracy: 0.9895\n",
            "Epoch 4/5\n",
            "238/238 [==============================] - 305s 1s/step - loss: 0.0373 - accuracy: 0.9883 - val_loss: 0.0305 - val_accuracy: 0.9882\n",
            "Epoch 5/5\n",
            "238/238 [==============================] - 298s 1s/step - loss: 0.0279 - accuracy: 0.9895 - val_loss: 0.0305 - val_accuracy: 0.9895\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J5Z1KPr_PUuR"
      },
      "source": [
        "# Make predictions with the model\n",
        "pred_probs = model.predict(test_df['text'].to_numpy())\n",
        "\n",
        "# Convert prediction probs to labels\n",
        "preds = tf.squeeze(tf.round(pred_probs))\n",
        "preds = tf.cast(preds , dtype = tf.int32)"
      ],
      "execution_count": 14,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "id": "XQ4zqP_KThLu",
        "outputId": "30e4da05-6225-41c1-94d8-19eb8df4e8aa"
      },
      "source": [
        "test_df"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>keyword</th>\n",
              "      <th>location</th>\n",
              "      <th>text</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Just happened a terrible car crash</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Heard about #earthquake is different cities, s...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>there is a forest fire at spot pond, geese are...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Apocalypse lighting. #Spokane #wildfires</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Typhoon Soudelor kills 28 in China and Taiwan</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3258</th>\n",
              "      <td>10861</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3259</th>\n",
              "      <td>10865</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Storm in RI worse than last hurricane. My city...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3260</th>\n",
              "      <td>10868</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>Green Line derailment in Chicago http://t.co/U...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3261</th>\n",
              "      <td>10874</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>MEG issues Hazardous Weather Outlook (HWO) htt...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3262</th>\n",
              "      <td>10875</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>#CityofCalgary has activated its Municipal Eme...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3263 rows × 4 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "         id keyword location                                               text\n",
              "0         0     NaN      NaN                 Just happened a terrible car crash\n",
              "1         2     NaN      NaN  Heard about #earthquake is different cities, s...\n",
              "2         3     NaN      NaN  there is a forest fire at spot pond, geese are...\n",
              "3         9     NaN      NaN           Apocalypse lighting. #Spokane #wildfires\n",
              "4        11     NaN      NaN      Typhoon Soudelor kills 28 in China and Taiwan\n",
              "...     ...     ...      ...                                                ...\n",
              "3258  10861     NaN      NaN  EARTHQUAKE SAFETY LOS ANGELES ÛÒ SAFETY FASTE...\n",
              "3259  10865     NaN      NaN  Storm in RI worse than last hurricane. My city...\n",
              "3260  10868     NaN      NaN  Green Line derailment in Chicago http://t.co/U...\n",
              "3261  10874     NaN      NaN  MEG issues Hazardous Weather Outlook (HWO) htt...\n",
              "3262  10875     NaN      NaN  #CityofCalgary has activated its Municipal Eme...\n",
              "\n",
              "[3263 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 419
        },
        "id": "1I6SFmqnUKwT",
        "outputId": "daebb67b-3859-4954-bd5d-0819a08967a2"
      },
      "source": [
        "submission = pd.DataFrame({'id': test_df['id'].values , \n",
        "                           'target': (preds.numpy())})\n",
        "submission"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>11</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3258</th>\n",
              "      <td>10861</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3259</th>\n",
              "      <td>10865</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3260</th>\n",
              "      <td>10868</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3261</th>\n",
              "      <td>10874</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3262</th>\n",
              "      <td>10875</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3263 rows × 2 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "         id  target\n",
              "0         0       1\n",
              "1         2       1\n",
              "2         3       1\n",
              "3         9       1\n",
              "4        11       1\n",
              "...     ...     ...\n",
              "3258  10861       1\n",
              "3259  10865       1\n",
              "3260  10868       1\n",
              "3261  10874       1\n",
              "3262  10875       1\n",
              "\n",
              "[3263 rows x 2 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7C6q3BNEURSw",
        "outputId": "ecb0e137-b678-44ad-9217-56735f376779"
      },
      "source": [
        "preds[:20]"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tf.Tensor: shape=(20,), dtype=int32, numpy=\n",
              "array([1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0],\n",
              "      dtype=int32)>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sDUlP6xZUocr"
      },
      "source": [
        "submission.to_csv('submission.csv' , index = False)"
      ],
      "execution_count": 18,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jtE8eZhoVMF8"
      },
      "source": [
        "Alright we have made a submission! \n",
        "\n",
        "![Screenshot 2021-07-09 at 1.59.07 PM.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UUfFI1TnYzPS"
      },
      "source": [
        "### 5. Combine the ensemble predictions using the majority vote (mode), how does this perform compare to averaging the prediction probabilities of each model?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xScdKAKgzL5I"
      },
      "source": [
        "- `model_5` -> Conv1D Model \n",
        "- `model_2` --> LSTM Model \n",
        "- `model` --> TFHub Model\n",
        "\n",
        "Now let's get the predictions for each model and perform majority model "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9R9NnDWRz8Qv",
        "outputId": "d7b90df4-8e31-4b66-c97b-f6937ffab741"
      },
      "source": [
        "test_sentences = test_df['text'].to_numpy()\n",
        "\n",
        "# Checking the shapes \n",
        "test_sentences.shape "
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(3263,)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uYHIPEqoY5gj"
      },
      "source": [
        "# Getting the model predictions \n",
        "\n",
        "def give_preds(model, test_sentences):\n",
        "  '''\n",
        "  This model takes a model and the test sentences as input. \n",
        "  Returns the prediction array of [0,1.....]\n",
        "  '''\n",
        "  pred_probs = model.predict(test_sentences)\n",
        "  preds = tf.squeeze(tf.round(pred_probs))\n",
        "  preds = tf.cast(preds , dtype = tf.int32)\n",
        "\n",
        "  return preds"
      ],
      "execution_count": 22,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EGnZNNVD06Lc"
      },
      "source": [
        "f1 , f2 and f3 are models \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "APSj-Iow3S4i",
        "outputId": "8d3159f6-c57c-4b46-daad-5e19e31796f5"
      },
      "source": [
        "# Getting the predictions for our 3 models \n",
        "model_preds = give_preds(model , test_sentences)\n",
        "model_2_preds = give_preds(model_2 ,test_sentences)\n",
        "model_5_preds = give_preds(model_5 , test_sentences)\n",
        "\n",
        "# Checking the shapes of our prediction \n",
        "model_preds.shape , model_2_preds.shape , model_5_preds.shape"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([3263]), TensorShape([3263]), TensorShape([3263]))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uO-8t4R83nHG",
        "outputId": "f8ccb711-5f86-4e06-f037-3496c9defba4"
      },
      "source": [
        "model_preds[:5] , model_2_preds[:5] , model_5_preds[:5]"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(<tf.Tensor: shape=(5,), dtype=int32, numpy=array([1, 1, 1, 1, 1], dtype=int32)>,\n",
              " <tf.Tensor: shape=(5,), dtype=int32, numpy=array([0, 0, 1, 0, 0], dtype=int32)>,\n",
              " <tf.Tensor: shape=(5,), dtype=int32, numpy=array([0, 0, 0, 0, 0], dtype=int32)>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "3vtL7zx98FUx",
        "outputId": "fe5d82e1-0924-43ef-e785-2ebd2afd893b"
      },
      "source": [
        "# Creating a dataframe of our predictions \n",
        "\n",
        "preds_df = pd.DataFrame({'USE_model': model_preds , \n",
        "              'LSTM_model': model_2_preds , \n",
        "              'Conv1D_model': model_5_preds})\n",
        "preds_df.head()"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>USE_model</th>\n",
              "      <th>LSTM_model</th>\n",
              "      <th>Conv1D_model</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   USE_model  LSTM_model  Conv1D_model\n",
              "0          1           0             0\n",
              "1          1           0             0\n",
              "2          1           1             0\n",
              "3          1           0             0\n",
              "4          1           0             0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "0k5sfhbT-Hf1",
        "outputId": "2b75c334-8ba6-46e2-92ca-9d4b5647b692"
      },
      "source": [
        "# Majority voted predictions of our model\n",
        "preds_df['majority_vote_preds'] = preds_df.mode(axis = 1)\n",
        "preds_df.head()"
      ],
      "execution_count": 48,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>USE_model</th>\n",
              "      <th>LSTM_model</th>\n",
              "      <th>Conv1D_model</th>\n",
              "      <th>majority_vote_preds</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   USE_model  LSTM_model  Conv1D_model  majority_vote_preds\n",
              "0          1           0             0                    0\n",
              "1          1           0             0                    0\n",
              "2          1           1             0                    1\n",
              "3          1           0             0                    0\n",
              "4          1           0             0                    0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fIXuv97W-YGw",
        "outputId": "697bc0ab-d269-4ecd-e751-220863f6fd67"
      },
      "source": [
        "# Converting the column into a numpy array \n",
        "mode_preds_array = preds_df['majority_vote_preds'].to_numpy()\n",
        "mode_preds_array"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0, 0, 1, ..., 0, 0, 0], dtype=int32)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EYpN3m4J_3oO"
      },
      "source": [
        "### 6. Make a confusion matrix with the best performing model's predictions on the validation set and the validation ground truth labels.\n",
        "\n",
        "Our best performing model is the `model` from the tensorflow hub. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pBs0bEVMAeZf"
      },
      "source": [
        "# Funtion taken from the course \n",
        "# Note: The following confusion matrix code is a remix of Scikit-Learn's \n",
        "# plot_confusion_matrix function - https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html\n",
        "import itertools\n",
        "import matplotlib.pyplot as plt\n",
        "import numpy as np\n",
        "from sklearn.metrics import confusion_matrix\n",
        "\n",
        "# Our function needs a different name to sklearn's plot_confusion_matrix\n",
        "def make_confusion_matrix(y_true, y_pred, classes=None, figsize=(10, 10), text_size=15, norm=False, savefig=False): \n",
        "  \"\"\"Makes a labelled confusion matrix comparing predictions and ground truth labels.\n",
        "\n",
        "  If classes is passed, confusion matrix will be labelled, if not, integer class values\n",
        "  will be used.\n",
        "\n",
        "  Args:\n",
        "    y_true: Array of truth labels (must be same shape as y_pred).\n",
        "    y_pred: Array of predicted labels (must be same shape as y_true).\n",
        "    classes: Array of class labels (e.g. string form). If `None`, integer labels are used.\n",
        "    figsize: Size of output figure (default=(10, 10)).\n",
        "    text_size: Size of output figure text (default=15).\n",
        "    norm: normalize values or not (default=False).\n",
        "    savefig: save confusion matrix to file (default=False).\n",
        "  \n",
        "  Returns:\n",
        "    A labelled confusion matrix plot comparing y_true and y_pred.\n",
        "\n",
        "  Example usage:\n",
        "    make_confusion_matrix(y_true=test_labels, # ground truth test labels\n",
        "                          y_pred=y_preds, # predicted labels\n",
        "                          classes=class_names, # array of class label names\n",
        "                          figsize=(15, 15),\n",
        "                          text_size=10)\n",
        "  \"\"\"  \n",
        "  # Create the confustion matrix\n",
        "  cm = confusion_matrix(y_true, y_pred)\n",
        "  cm_norm = cm.astype(\"float\") / cm.sum(axis=1)[:, np.newaxis] # normalize it\n",
        "  n_classes = cm.shape[0] # find the number of classes we're dealing with\n",
        "\n",
        "  # Plot the figure and make it pretty\n",
        "  fig, ax = plt.subplots(figsize=figsize)\n",
        "  cax = ax.matshow(cm, cmap=plt.cm.Blues) # colors will represent how 'correct' a class is, darker == better\n",
        "  fig.colorbar(cax)\n",
        "\n",
        "  # Are there a list of classes?\n",
        "  if classes:\n",
        "    labels = classes\n",
        "  else:\n",
        "    labels = np.arange(cm.shape[0])\n",
        "  \n",
        "  # Label the axes\n",
        "  ax.set(title=\"Confusion Matrix\",\n",
        "         xlabel=\"Predicted label\",\n",
        "         ylabel=\"True label\",\n",
        "         xticks=np.arange(n_classes), # create enough axis slots for each class\n",
        "         yticks=np.arange(n_classes), \n",
        "         xticklabels=labels, # axes will labeled with class names (if they exist) or ints\n",
        "         yticklabels=labels)\n",
        "  \n",
        "  # Make x-axis labels appear on bottom\n",
        "  ax.xaxis.set_label_position(\"bottom\")\n",
        "  ax.xaxis.tick_bottom()\n",
        "\n",
        "  ### Added: Rotate xticks for readability & increase font size (required due to such a large confusion matrix)\n",
        "  plt.xticks(rotation=70, fontsize=text_size)\n",
        "  plt.yticks(fontsize=text_size)\n",
        "\n",
        "  # Set the threshold for different colors\n",
        "  threshold = (cm.max() + cm.min()) / 2.\n",
        "\n",
        "  # Plot the text on each cell\n",
        "  for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "    if norm:\n",
        "      plt.text(j, i, f\"{cm[i, j]} ({cm_norm[i, j]*100:.1f}%)\",\n",
        "              horizontalalignment=\"center\",\n",
        "              color=\"white\" if cm[i, j] > threshold else \"black\",\n",
        "              size=text_size)\n",
        "    else:\n",
        "      plt.text(j, i, f\"{cm[i, j]}\",\n",
        "              horizontalalignment=\"center\",\n",
        "              color=\"white\" if cm[i, j] > threshold else \"black\",\n",
        "              size=text_size)\n",
        "\n",
        "  # Save the figure to the current working directory\n",
        "  if savefig:\n",
        "    fig.savefig(\"confusion_matrix.png\")"
      ],
      "execution_count": 50,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LQNUJ8AaBGlx",
        "outputId": "553b1ca1-6aac-4551-b469-3df8412411a3"
      },
      "source": [
        "# Predictions on the validation sentences \n",
        "val_preds = give_preds(model , val_sentences)\n",
        "\n",
        "# Creating a array of our class names \n",
        "class_names = ['not_a_disaster' , 'disaster']\n",
        "\n",
        "# Checking the preds and true labels shape\n",
        "val_preds.shape , val_labels.shape"
      ],
      "execution_count": 57,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(TensorShape([762]), (762,))"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 576
        },
        "id": "OhQr5Xf0BV9o",
        "outputId": "80e7d825-9342-4371-f04c-b1e7be2f2c5c"
      },
      "source": [
        "# Plotting the confusion matrix \n",
        "make_confusion_matrix(y_true= val_labels , \n",
        "                      y_pred = val_preds , \n",
        "                      classes = class_names , \n",
        "                      figsize= (15 , 7), \n",
        "                      norm = False , \n",
        "                      savefig = True , \n",
        "                      text_size = 15)"
      ],
      "execution_count": 61,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmQAAAIvCAYAAADajscTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOzdeZwdVZXA8d/JngBJgLCEHcewI4sZNkWRHQQJDiCIiAhGBFTGfdRBREDBDRBFQVFAWRWUHR2WEVCQHdkJ24QQlkBYEwKJZ/6oanhpkl7C665XL78vn/rk1a16Vfd1OvTpc+69FZmJJEmSqjOg6g5IkiQt7AzIJEmSKmZAJkmSVDEDMkmSpIoZkEmSJFXMgEySJKliBmSSaiUihkfERRHxQkSc9zaus3dE/LmZfatCRFwWEftW3Q9Jb48BmaQ+EREfjYibI+LliJhaBg7vbcKldwOWAZbMzN0X9CKZ+bvM3LYJ/ZlLRGwRERkRF3RqX69sv6aH1zk8In7b3XmZuUNmnraA3ZXUIgzIJDVdRHwBOA44miJ4Wgn4GbBLEy6/MvBAZs5uwrX6yjPAphGxZEPbvsADzbpBFPx/uNQm/McsqakiYhRwBHBwZp6fma9k5uuZeVFmfrk8Z2hEHBcRT5TbcRExtDy2RUQ8HhFfjIiny+zafuWxbwOHAR8pM2/7d84kRcQqZSZqULn/iYh4OCJeiohHImLvhvbrGt63WUTcVJZCb4qIzRqOXRMR34mI68vr/DkixnTxZXgN+COwZ/n+gcBHgN91+lodHxGTI+LFiLglIjYv27cHvt7wOe9o6MdREXE9MAN4R9l2QHn8pIj4Q8P1j4mIKyMievwXKKkSBmSSmm1TYBhwQRfnfAPYBFgfWA/YCPhmw/FlgVHA8sD+wE8jYvHM/BZF1u2czFw0M3/VVUciYhHgBGCHzFwM2Ay4fR7nLQFcUp67JPAj4JJOGa6PAvsBSwNDgC91dW/gdODj5evtgLuAJzqdcxPF12AJ4EzgvIgYlpmXd/qc6zW8Zx9gIrAY8Fin630RWLcMNjen+Nrtmz4jT2p5BmSSmm1JYFo3JcW9gSMy8+nMfAb4NkWg0eH18vjrmXkp8DKw+gL251/AOhExPDOnZubd8zjng8CDmXlGZs7OzLOA+4CdG875dWY+kJkzgXMpAqn5ysy/AUtExOoUgdnp8zjnt5n5bHnPHwJD6f5z/iYz7y7f83qn682g+Dr+CPgt8NnMfLyb60lqAQZkkprtWWBMR8lwPpZj7uzOY2XbG9foFNDNABbtbUcy8xWKUuGBwNSIuCQi1uhBfzr6tHzD/pML0J8zgEOADzCPjGFEfCki7i3LpM9TZAW7KoUCTO7qYGbeCDwMBEXgKKkGDMgkNdvfgVnAhC7OeYJicH6HlXhrOa+nXgFGNOwv23gwM6/IzG2AsRRZr1N60J+OPk1ZwD51OAM4CLi0zF69oSwpfgXYA1g8M0cDL1AEUgDzKzN2WX6MiIMpMm1PlNeXVAMGZJKaKjNfoBh4/9OImBARIyJicETsEBHHlqedBXwzIpYqB8cfRlFiWxC3A++LiJXKCQX/1XEgIpaJiF3KsWSzKEqf/5rHNS4FViuX6hgUER8B1gIuXsA+AZCZjwDvpxgz19liwGyKGZmDIuIwYGTD8aeAVXozkzIiVgOOBD5GUbr8SkR0WVqV1BoMyCQ1XTke6gsUA/WfoSizHUIx8xCKoOFm4E7gn8CtZduC3OsvwDnltW5h7iBqQNmPJ4DnKIKjz8zjGs8CO1EMin+WIrO0U2ZOW5A+dbr2dZk5r+zfFcDlFEthPAa8ytzlyI5Fb5+NiFu7u09ZIv4tcExm3pGZD1LM1DyjYwarpNYVTr6RJEmqlhkySZKkihmQSZIkVcyATJIkqWIGZJIkSRUzIJMkSaqYAZkkSVLFDMgkSZIqZkAmSZJUMQMySZKkihmQSZIkVcyATJIkqWIGZJIkSRUzIJMkSaqYAZkkSVLFDMgkSZIqZkAmSZJUMQMySZKkihmQSZIkVcyATJIkqWIGZJIkSRUzIJMkSaqYAZkkSVLFDMgkSZIqZkAmSZJUsUFVd0CtLwYNzxiyWNXdkPrVBmuuVHUXpErceust0zJzqar70Z2BI1fOnD2zKdfKmc9ckZnbN+ViC8iATN2KIYsxdPU9qu6G1K+uv/HEqrsgVWL44His6j70RM6e2bSfTa/e/tMxTbnQ22BAJkmSaigg2mfkVft8EkmSpJoyQyZJkuongIiqe9E0BmSSJKmeLFlKkiSpWcyQSZKkerJkKUmSVCVnWUqSJFUvojlbj24VAyPitoi4uNxfNSJujIhJEXFORAwp24eW+5PK46v05PoGZJIkSd37PHBvw/4xwI8z853AdGD/sn1/YHrZ/uPyvG4ZkEmSpPoJipJlM7bubhWxAvBB4JflfgBbAr8vTzkNmFC+3qXcpzy+VXl+lwzIJElSDTWpXNmzkuVxwFeAf5X7SwLPZ+bscv9xYPny9fLAZIDy+Avl+V0yIJMkSQu7MRFxc8M2seNAROwEPJ2Zt/RlB5xlKUmS6ql5syynZeb4+Rx7D/ChiNgRGAaMBI4HRkfEoDILtgIwpTx/CrAi8HhEDAJGAc921wEzZJIkqZ76oWSZmf+VmStk5irAnsBVmbk3cDWwW3navsCfytcXlvuUx6/KzOzuoxiQSZIk9d5XgS9ExCSKMWK/Ktt/BSxZtn8B+FpPLmbJUpIk1VD/LwybmdcA15SvHwY2msc5rwK79/baBmSSJKl+grZ6dJIlS0mSpIqZIZMkSfXURs+yNCCTJEk15MPFJUmS1ERmyCRJUj0NaJ9B/QZkkiSpfjoeLt4m2ueTSJIk1ZQZMkmSVE9ttA6ZAZkkSaohZ1lKkiSpicyQSZKkerJkKUmSVDFLlpIkSWoWM2SSJKl+IixZSpIkVc6SpSRJkprFDJkkSaonS5aSJElVcmFYSZIkNZEZMkmSVE+WLCVJkioUWLKUJElS85ghkyRJNdReg/oNyCRJUj210Riy9gktJUmSasoMmSRJqidLlpIkSRWzZClJkqRmMUMmSZLqJ5xlKUmSVD1LlpIkSWoWM2SSJKmWoo0yZAZkkiSpdoL2CsgsWUqSJFXMDJkkSaqfKLc2YUAmSZJqKCxZSpIkLSwiYlhE/CMi7oiIuyPi22X7byLikYi4vdzWL9sjIk6IiEkRcWdEbNjdPcyQSZKkWurHDNksYMvMfDkiBgPXRcRl5bEvZ+bvO52/AzCu3DYGTir/nC8DMkmSVEv9FZBlZgIvl7uDyy27eMsuwOnl+26IiNERMTYzp87vDZYsJUmSuhERAyPiduBp4C+ZeWN56KiyLPnjiBhati0PTG54++Nl23wZkEmSpFqKiKZswJiIuLlhm9j5Xpk5JzPXB1YANoqIdYD/AtYA/h1YAvjqgn4WS5aSJKl+mrvsxbTMHN+TEzPz+Yi4Gtg+M39QNs+KiF8DXyr3pwArNrxthbJtvsyQSZIkdSEiloqI0eXr4cA2wH0RMbZsC2ACcFf5lguBj5ezLTcBXuhq/BiYIZMkSTUU/bsO2VjgtIgYSJHMOjczL46IqyJiKYpc3e3AgeX5lwI7ApOAGcB+3d3AgEySJNVSP86yvBPYYB7tW87n/AQO7s09DMgkSVItuVK/JEmSmsYMmSRJqqV2ypAZkEmSpPpp7rIXlbNkKUmSVDEzZJIkqZYsWUqSJFWon9ch63OWLCVJkipmhkySJNVSO2XIDMgkSVI9tU88ZslSkiSpambIJElS/YQlS0mSpMq1U0BmyVKSJKliZsgkSVIttVOGzIBMkiTVjgvDSpIkqanMkEmSpHpqnwSZAZkkSaqhNlv2wpKlVJHllhrFM9f/kJm3ncgiw4e80T5x9805/4QDefzqY5h524ls/u5xC3QdqW7uvecedth2K5YYOYJVV1qOIw4/jDlz5lTdLalfGJBJFTn6P3fl5Rmz3tK+904bscSoRfifv9/7tq4j1cn06dPZcfutiQjOO/9PfP0bh3H8j3/Id779raq7phYWEU3ZWoEBmVSB92z4b2yz2Zocf/qVbzm2xSd+xBb7/pBjf3XF27qOVCe/PPnnvDpzJmefdz5bbb0Nn/r0gXz9v7/FCcf9iBdffLHq7qlFGZBJWmADBgQ/+urufPfky5j2/MtvOZ6ZTbmOVCdXXH4ZW2+7HSNHjnyjbfc99mTmzJlc+9f/rbBnUv8wIJP62ad225yhgwfx83P/2hLXkVrBA/ffx+qrrzFX20orrcSIESO4//77KuqVWl40aWsBzrKU+tESoxbhsIM+yCe/eRqzZ/+r8utIrWL69OmMGjX6Le2jF1+c56dPr6BHqoNWKTc2gxkyqR8dfsjO/OOfj3LFdfe0xHUkSa2hFgFZREyMiAlV9wMgItaJiIyILRraMiIO6ed+tMzXRD2z5juWZd9dNuG7J1/GqEWHM2rR4YwYVixTMWrR4QwbOrhfryO1ksUXX5wXX3zhLe3PT5/O6MUXr6BHanXNGtDfKlm2upQsJwJ3AX+suiPzsSnwSD/fs9W/JurknSstzZDBg/jf07/0lmMP/fkofn3B3zjoiDP77TpSK1lt9TXeMlZs8uTJzJgx4y1jy6QOrRJMNUNdArKWlpk3VN2HtyMiBgIDM/O1qvvSzv52+0Nse8Dxc7Vt+541+dJ+27LLIT/jkcen9et1pFay3fY78OMffp+XXnqJxRZbDIDfn3cOw4cPZ/P3vb/i3kl9r09LlhHxm4i4OSK2iYg7I+KViLguItZuOGdERJwQEU9GxKsRcVNEbNtw/Brg3cC+ZWkwI+ITPbj32Ig4NSIejoiZEfFARBwZEb1ayjwiDoqIyWXfLwLGzuOcuUqWEfHeiLg2Il4st9sjYveG4x8vvw7PRcT0iLg6IsZ3uubaEXF5ec4rEXFvRBzck69JRBwQEXdHxKyIeCwivtLp2h1/LxMi4m7gVWDj3nxd1HvPPv8K197y4Fzb/Y88BcD1t07iwceeBmDDtVZi163XZ6tNiqzA5u9+J7tuvT4brrVSr64j1ckBEw9k6NCh7Ln7h7nqyv/hV6eczFFHHM7nDv3CXEthSI0sWfbOSsD3gaOAmcAPgHMiYt0sFlw6BfgQ8HVgEvAp4JKI+EBmXgccBPwBeBj4TnnNh3pw3zHAc8AXgOnAasDhwFLAp3vS8YjYBfgp8HOK0uD7gVO7ec9I4GLgT8ARFBNq1wUapw+tApxefo4hwF7AtRGxdmY+XJ5zEXAv8DFgFrA60PF/pfl+TSLiy8DRwLHANRSB23ciYkZmntipD8eWfXyS/i+5aj4O/Mj72OdDm7yx/9+f+SAAZ1x4AxO/9duquiX1qcUXX5xLr7iS//z8IfzHhJ0ZPXo0n/38f/LNww6vumtqZa0RSzVF9HQRygW6eMRvKAKKNTPzwbJtAnABsCbFl/JuYL/MPK08PgC4E5iSmduVbTcDd2XmJ95GXwYBe1AEVCN7Up6LiH8Az2bmDg1tpwAHAB/IzGvKtgQ+m5knlpmum8p7vNSDewygyFTeBZyZmUdExBjgGeBdmfnP+bzvLV+TMhh8Avh+Zn67of0IijFny2fmnPLvZV9gg8y8fT7Xn1i+BwYv+u5ha+/b3UeR2sr0m07s/iSpDQ0fHLdk5vjuz6zW0GXG5XIfPa4p13r0uJ0q/8z9Mcvy0Y5grNQxT38F4N8pgrLzOg5m5r/K/fe+nZtG4dCIuCciZgKvA78DhlJk7bp7/yBgQ4pMV6Pzu3nrQ8DLwJkRsUtEvGVhnYhYMyIuiIingDll31anyOJBkdmbDPw8Ij4SEUt319/SpsAiwHkRMahjA64ClqH4mneYMr9gDCAzT87M8Zk5PgYN7+HtJUnqP+1UsuyPgOz5TvsdmalhFOOxXs7MGZ3OeQoYERFD38Z9D6Uoj14A7AJsBBzccO/ujAEGAp0H5HQ5QCczpwPbAIOBc4FnIuKSiHgHQEQsBvwZWJGinLo5RWB6R0e/yqB0W4pS4qnAk+WYtA160Gcoso6vN2xXl+0rNpz7VDfXkiSpdUV7BWRVz7KcCiwaESM6BWXLADMyc9bbuPbuwO8z8xsdDRGxVi/eP40ie9U5O9Vttqqcdbl9RAwHtgZ+BJwJbEKRxVoB2CYz35jjHRGjOl3jPuA/ImIwRdB2DMXYuhXKgG1eniv/3Il5B1z3N96iu88hSZL6R9ULw95EERjs1tEQRai6G3Bdw3mv0bOsVqPhFIPhG+3d0zdn5mzgNorsWqMP9+IaMzPzIoosV0cw2FH/e6NvEbEZxSD7eV3j9cy8iiKoG8ubkwPm9TX5O8XEieUy8+Z5bN2OaZMkqQ4CiGjO1goqzZBl5r0RcRZwYlnKe4hiluUawGcaTr0P2C4itgOeBR7JzGe7ufxfgM9FxI3ldfcG3tnLLh4NnB8RJ1GUPt8PbN/VGyLig8AnKWZl/h+wPMWszqvKU26gGGN2SkQcS5EtOxyY0nCNd1HORqWYSbk48FXgjszsyILN82sSEYcDx0fEysBfKYLu1SgmIezay88vSVKLap1yYzNUnSGDIgA7DTiMYgD9ysBO5ZIXHY6kWALiXIqs2s49uO4RwFnle8+iyCh9rjcdy8wLgM+W9/sjsAGwfzdvm0SR9TuaYqzYscDlFEEamfkURTl1WYrPeyhwYPm+Dk9SlBy/AVwG/Izi83+o4Zx5fk0y81iK2ZE7lNc/iyIYvbY3n12SJBUiYlhE/CMi7ohinc9vl+2rRsSNETEpIs6Jcq3TiBha7k8qj6/S7T36ctkLtYcBI5bOoavvUXU3pH7lshdaWNVl2Ythy66WK338hKZc68Hv79DlZy6HUy2SmS+XY7uvAz5PMTnv/Mw8OyJ+TlHJOikiDqJYuurAiNgT2DUzP9JVH1ohQyZJktRr/TXLMgsvl7uDyy2BLYHfl+2nARPK17uU+5THt4publT1LMsFUn6ogV2cMie7Sf3FmwuyzlM5qF+SJIkonvt8C8V49J9SjE9/viFeeJxi3Djln5OhiCci4gVgSYoVHOaprhmyfZl7na3OW0+WlT+1q2v0pN4rSZIq0qQZlmXeakwUz3ju2CZ2vl1mzsnM9Skm421EMQGxaWqZIaN4zuO/d3G8J89lPBzoapDIE73pkCRJ6j8BDBjQtFmW03o6bi4zn4+IqynWFR0dEYPKLNkKvLliwhSKxdgfL5+YM4piRYT5qmVAVi550d2yF91d41Hg0Wb0R5Ikta+IWAp4vQzGhlM8kecYiifh7AacTVGd63jc4oXl/t/L41d1N5SqlgGZJElSPy5DNhY4rRxHNgA4NzMvjoh7gLMj4kiKxeR/VZ7/K+CMiJhE8RSdPbu7gQGZJEmqpf5aGDYz76RYi7Rz+8MU48k6t79KseZoj9V1UL8kSVLbMEMmSZLqp4WeQ9kMBmSSJKl2ioeLt09EZslSkiSpYmbIJElSDfXssUd1YUAmSZJqqY3iMUuWkiRJVTNDJkmSasmSpSRJUpVc9kKSJKlaLnshSZKkpjJDJkmSaqmNEmQGZJIkqZ4sWUqSJKlpzJBJkqRaaqMEmQGZJEmqobBkKUmSpCYyQyZJkmqnWIes6l40jwGZJEmqobBkKUmSpOYxQyZJkmqpjRJkBmSSJKmeLFlKkiSpacyQSZKk+glLlpIkSZUqlr1on4jMkqUkSVLFzJBJkqRaaqcMmQGZJEmqpTaKxyxZSpIkVc0MmSRJqiVLlpIkSVVqs2UvLFlKkiRVzAyZJEmqnSAsWUqSJFWtjeIxS5aSJElVM0MmSZJqaUAbpcgMyCRJUi21UTxmyVKSJKkrEbFiRFwdEfdExN0R8fmy/fCImBIRt5fbjg3v+a+ImBQR90fEdt3dwwyZJEmqnYh+XRh2NvDFzLw1IhYDbomIv5THfpyZP5i7b7EWsCewNrAc8D8RsVpmzpnfDQzIJElSLQ3op3gsM6cCU8vXL0XEvcDyXbxlF+DszJwFPBIRk4CNgL/P7w2WLCVJknooIlYBNgBuLJsOiYg7I+LUiFi8bFsemNzwtsfpOoAzIJMkSfUUEU3ZgDERcXPDNnE+91sU+ANwaGa+CJwE/BuwPkUG7YcL+lksWUqSpFpq4hCyaZk5vut7xWCKYOx3mXk+QGY+1XD8FODicncKsGLD21co2+bLDJkkSVIXokij/Qq4NzN/1NA+tuG0XYG7ytcXAntGxNCIWBUYB/yjq3uYIZMkSbUTFM+z7CfvAfYB/hkRt5dtXwf2ioj1gQQeBT4NkJl3R8S5wD0UMzQP7mqGJRiQSZKkmurHWZbXwTyjv0u7eM9RwFE9vYclS0mSpIqZIZMkSfXz5gzJtmBAJkmSaqmN4jFLlpIkSVUzQyZJkmongAFtlCIzIJMkSbXURvGYJUtJkqSqmSGTJEm15CxLSZKkCkVYspQkSVITmSGTJEm15CxLSZKkirVPONZFQBYRP6F4evk8Zebn+qRHkiRJC5muMmQ391svJEmSemmhmGWZmac17kfEiMyc0fddkiRJ6lqxUn/VvWiebmdZRsSmEXEPcF+5v15E/KzPeyZJkrSQ6MmyF8cB2wHPAmTmHcD7+rJTkiRJXYogmrS1gh7NsszMyZ06PKdvuiNJktQzLRJLNUVPArLJEbEZkBExGPg8cG/fdkuSJKlrrZLdaoaelCwPBA4GlgeeANYv9yVJktQE3WbIMnMasHc/9EWSJKlHFsZZlu+IiIsi4pmIeDoi/hQR7+iPzkmSJM1POw3q70nJ8kzgXGAssBxwHnBWX3ZKkiRpYdKTgGxEZp6RmbPL7bfAsL7umCRJUleiSVsr6OpZlkuULy+LiK8BZ1M82/IjwKX90DdJkqR5ioABLVJubIauBvXfQhGAdXzaTzccS+C/+qpTkiRJC5OunmW5an92RJIkqTfaKEHWs5X6I2IdYC0axo5l5ul91SlJkqTutMoMyWboNiCLiG8BW1AEZJcCOwDXAQZkkiRJTdCTWZa7AVsBT2bmfsB6wKg+7ZUkSVI3IpqztYKelCxnZua/ImJ2RIwEngZW7ON+SZIkzVcQC80syw43R8Ro4BSKmZcvA3/v015JkiQtRHryLMuDypc/j4jLgZGZeWffdkuSJKkLLVRubIauFobdsKtjmXlr33RJkiSpewvLLMsfdnEsgS2b3Be1qPXXXIm//u2Eqrsh9avl9/eRvZL6T1cLw36gPzsiSZLUGz1ZKqIuerQwrCRJUisJ2qtk2U7BpSRJUtNFxIoRcXVE3BMRd0fE58v2JSLiLxHxYPnn4mV7RMQJETEpIu7salx+BwMySZJUSwOiOVsPzAa+mJlrAZsAB0fEWsDXgCszcxxwZbkPxVONxpXbROCkbj9LdyeUUd7HIuKwcn+liNioR92XJEnqI/0VkGXm1I7VJTLzJeBeYHlgF+C08rTTgAnl612A07NwAzA6IsZ2+Vl68Hl/BmwK7FXuvwT8tAfvkyRJaisRsQqwAXAjsExmTi0PPQksU75eHpjc8LbHy7b56smg/o0zc8OIuA0gM6dHxJCed12SJKm5iudQNm1Q/5iIuLlh/+TMPPmt94xFgT8Ah2bmi433z8yMiFzQDvQkIHs9IgZSrD1GRCwF/GtBbyhJktQMPRz/1RPTMnN8VydExGCKYOx3mXl+2fxURIzNzKllSfLpsn0Kcz/3e4Wybb56UrI8AbgAWDoijgKuA47uwfskSZJqL4pU2K+AezPzRw2HLgT2LV/vC/ypof3j5Tj8TYAXGkqb89STZ1n+LiJuAbaiWPZjQmbe27uPIkmS1Fz9uAzZe4B9gH9GxO1l29eB7wHnRsT+wGPAHuWxS4EdgUnADGC/7m7QbUAWESuVF7uosS0z/6/nn0OSJKl5AhjQTxFZZl5X3nJetprH+Qkc3Jt79GQM2SUU48cCGAasCtwPrN2bG0mSJGneelKyXLdxv1xt9qA+65EkSVIPtNPq9r1+lmVm3hoRG/dFZyRJknqqjR5l2aMxZF9o2B0AbAg80Wc9kiRJWsj0JEO2WMPr2RRjyv7QN92RJEnqXkT026D+/tBlQFYuCLtYZn6pn/ojSZLUI20Uj81/PFxEDMrMORRrb0iSJKmPdJUh+wfFeLHbI+JC4DzglY6DDY8NkCRJ6ndNfHRS5XoyhmwY8CywJW+uR5aAAZkkSapEfy4M2x+6CsiWLmdY3sWbgViHBX6auSRJkubWVUA2EFiUeT8qwIBMkiRVqo0SZF0GZFMz84h+64kkSVJPRXuNIevqqQNt9DElSZJaV1cZsrc8vVySJKlVRBvljuYbkGXmc/3ZEUmSpJ4qZllW3YvmaacHpUuSJNVST9YhkyRJajntlCEzIJMkSbUUbbTuhSVLSZKkipkhkyRJtdNug/oNyCRJUv1Ee63Ub8lSkiSpYmbIJElSLQ1ooxSZAZkkSaqddhtDZslSkiSpYmbIJElSLbVRxdKATJIk1VEwYGF4uLgkSVKrCtorQ+YYMkmSpIqZIZMkSfUT7TXL0oBMkiTVUjutQ2bJUpIkqWJmyCRJUu2026B+AzJJklRLliwlSZLUNGbIJElSLbVRgsyATJIk1U/QXmW+dvoskiRJTRcRp0bE0xFxV0Pb4RExJSJuL7cdG479V0RMioj7I2K7ntzDDJkkSaqfgOi/muVvgBOB0zu1/zgzfzBXtyLWAvYE1gaWA/4nIlbLzDld3cAMmSRJqqVo0tadzPwr8FwPu7ULcHZmzsrMR4BJwEbdvcmATJIkacEcEhF3liXNxcu25YHJDec8XrZ1yYBMkiTVTlCsQ9aMDRgTETc3bBN70IWTgH8D1gemAj98O5/HMWSSJKmWmjiCbFpmju/NGzLzqTf6EXEKcHG5OwVYseHUFcq2LpkhkyRJ6qWIGNuwuyvQMQPzQmDPiBgaEasC44B/dHc9M2SSJKmW+muSZUScBWxBUdp8HPgWsEVErA8k8CjwaYDMvDsizgXuAWYDB3c3wxIMyCRJUi1Fvy17kZl7zaP5V12cfxRwVG/uYclSkiSpYmbIJElS7bTbo5MMyCRJUi3140r9fa6dgktJkqRaMkMmSZJqqX3yYwZkkiSpjvr34eJ9zpKlJElSxcyQSeYU8+4AACAASURBVJKk2nGWpSRJUguwZClJkqSmMUMmSZJqqX3yYwZkkiSpptqoYmnJUpIkqWpmyCRJUu0UsyzbJ0VmQCZJkmrJkqUkSZKaxgyZJEmqoSAsWUqSJFXLkqUkSZKaxgyZJEmqHWdZSpIkVS0sWUqSJKmJzJBJkqRaaqcMmQGZJEmqpXZa9sKSpSRJUsXMkEmSpNoJYED7JMjMkEmt6Pfnns17NxnPskuOZLV3rMjET+7L1CeeqLpb0gLZefyKXPbNrXnwpx9myil7cOP3PsgXP7Q2gwfO+0fQkR/dgGdP24tv77n+W46tvtxILvjKB5h88u7cfdwufG3XdRnQTgOJ1CvRpP9agRkyqcVccvGF7PfxvZl44EEcefQxPPnkVL5z+GHstuvOXPv3mxgwwN+jVC9LLDqEa+99ihMvu48XXnmNDd+xJF/ZdR2WHjWMr55xy1znrr7cSD72vn/jxRmvveU6o0YM5vyvfID7n3iRfY6/llWWXpQj9tqAAQPg6D/8s78+jtQnDMikFnPeOWex/gYb8sPjfvJG22IjR7LnbrvywAP3s8Yaa1bYO6n3Trvmobn2r7vvaRYbPpj9txr3loDsex97N7/48/3ssdkqb7nOfluOY9iQQex7wrW89OpsuBsWGz6Yr0xYh59ccm/RpoVKOyVH/VVbajGvv/46I0eOmqtt1KjRxYvMCnokNd9zL89i8KC5fwTtPH5Fxi03kuMvvmee79nqXWO56p9T5wq8zr/hMUYMHcRmayzdp/1Va2qnkqUBmdRi9tl3P/52/bWc+dvTefHFF3nwwQf4zuGH8f4ttmSNNdequnvSAhsQwfAhA9l43BgmbrMav75q0hvHhg0eyHf22oAjzr2DGa/Nmef7x40dyYNTX5yrbcpzM3hl1mzGjR3Zp32X+polS6nFbL/DB/n5Kady8IGf4tMH7AfAxptsxjl/+GPFPZPenskn786wIQMBOPu6R/jWObe9cezQndbiqedncu7fHp3v+0ePGMILM15/S/sLr7zG6EWGNL2/am3OshQRsU5EZERsUe5nRBzSz32YGBET+vOe6h9/veZqDv3sQXzmkM9x6RVX8pszzmT69Of46B7/wZw5884cSHWww5F/Yccj/8I3z7yVHTZYnmP3GQ/ASmMW4eAd1uDrv7u14h6qXppVsGyNqM4MWXNsCjzSz/ecCNwFmDZpM1//2pfZ8YM7852jvvdG27rrrc+737UWF1/0J3aZ8OEKeyctuDsfmw7AjQ9O47mXZ/GziZvy08vv45u7vYsr75zKg0++yMgRgwEYMCAYOmggI0cM5sUyK/b8jNfeON5o1CJDeP6Vt87KlOrEgKwJMvOGqvvwdkTEQGBgZvp/tBbwwP33sdsee87VttpqqzN8+HAeefjhinolNdcdjxbB2cpjFuGdy45k3ZUXZ+d/X3Gucz61zWp8apvVWPfQP/LE9Jk8OPXFt4wVW26JESwydNBbxpZpIRDOslzoRMRBETE5Il6JiIuAsZ2Oz1WyjIj3RsS1EfFiud0eEbs3HP94RFwXEc9FxPSIuDoixne65toRcXl5zisRcW9EHFweuwZ4N7Bvee+MiE80vPeAiLg7ImZFxGMR8ZVO1/5NRNwcERMi4m7gVWDjpn3B9LasuNLK3HHb3KWb++67l5kzZ7LyyitX1CupuTYeNwaAx6a9wqGn/oMPfffKubannp/JBTc+xoe+eyXTXpoFwJV3TmXLdZZl0WFv5hJ23XglZsyazd/ue7qSz6FqRZO2VmCGrBsRsQvwU+DnFOXB9wOndnH+SOBi4E/AERR/1+sCoxtOWwU4HXgIGALsBVwbEWtnZkcK5CLgXuBjwCxgdaDjV8ODgD8ADwPfKdseKu//ZeBo4FjgGorA7TsRMSMzT+zUh2PLPj5J/5dcNR/7f+rTfO3LX2DZ5ZZj22235+mnn+J7Rx/Jyiuvwrbb71h196ReO/eLW/C/9zzJfVNeYM6/ko3HjeGg7dfg/Bse49GnX57ne2a9Pocpz83g+oZA69dXPcintlmN0z67OSdccg8rL70oX5mwDiddcZ9rkC2EikH9rRJOvX0GZN37BnB5Zn6m3L8iIpYCDpjP+asBo4BDMvOlsu3PjSdk5hEdryNiAPAXYCOK4OuIiBgDrArskpkdy09f2fD+eyLiFeCZxnJpGQx+CzgyM79dNv8lIkYA34yIkzKzY1T4ksDWmXn7vD5EREykGKfGiiuuNJ+Pqr7wmYM/y5Ahg/nlyb/g1FN+wajRo9l0s/dw+BFHs8gii1TdPanXbnvkWfZ676qsOGYR5sxJHn3mZY487w5+ffWk7t/c4IUZr/PhY67imH3G87v/fB8vzHidn19xP8dccFcf9VwqRMSpwE7A05m5Ttm2BHAORYLjUWCPzJweEQEcD+wIzAA+kZndzlgxIOtCRAwCNgQ6z6A8n/kHZA8BLwNnRsQvgf/NzOc7XXdNiizWZkDjaoarlX8+B0wGfh4RJwBXZ2ZP8vGbAosA55V973AV8N/ACsBjZduU+QVjAJl5MnAywIbvHu9qpP0oIjhg4mc4YOJnuj9ZqoHvnv9Pvnt+7x5ttMGXLppn+/1PvMiEY65qRrfUBvoxP/Yb4ESK6laHrwFXZub3IuJr5f5XgR2AceW2MXASPRgW5Biyro0BBgKdg6H5BkeZOR3YBhgMnAs8ExGXRMQ7ACJiMYqM2YrAF4DNgX8H7gCGldf4F7AtRSnxVODJckzaBj3oL8DdwOsN29Vle+OI2ae6uZYkSa2tnwaRZeZfKZIljXYBTitfnwZMaGg/PQs3AKMjYizdMEPWtWnAHObOYjGP/bmUfwHbR8RwYGvgR8CZwCYUWawVgG0y876O90TEqE7XuA/4j4gYTBG0HQNcEhErlAHbvHR8s+zEvAOu+xtv0dVnkCRJXVomM6eWr58ElilfL09R5erweNk2lS6YIetCZs4GbqOIdhv1aCGozJyZmRdRZLk6nnkzvPxzVsd5EbEZRQ16Xtd4PTOvogjqxvLm5IDXKDNqDf4OzASWy8yb57G9hCRJbaKJC8OOKVcf6Ngm9qYfmZm8zUSHGbLuHQ2cHxEnARdQzLLcfn4nR8QHgU9SzMj8P4qo+NMU47gAbqAYY3ZKRBxLkS07HJjScI13AT+gGCz4MLA4RV36jszsyILdB2wXEdsBzwKPZOazEXE4cHxErAz8lSLoXg34QGbu+ra+EpIktZAmTrKclpnjuz9tLk9FxNjMnFqWJDuGM01h7iFCK9DwM35+zJB1IzMvAD4L7EwRZG0A7N/FWyZRRMlHU4wVOxa4nCJIIzOfAnYHlqVYGuNQ4MDyfR2epCg5fgO4DPgZxRIYH2o458iy7VzgprJ/ZOaxFLMjdyivfxawN3Bt7z+9JEmajwuBfcvX+1L8zO1o/3gUNgFeaChtzpcZsh4o1+86sVNzNBxvfH0/sFs317ucIkhrdGnD8aeBfbq5xsMU49Pmdey3wG+7eO8nurq2JEl10F+zLCPiLGALitLm4xRLTH0PODci9qdYwWCP8vRLKZa8mESx7MV+PbmHAZkkSaqnforIMnOv+Rzaah7nJnBwb+9hyVKSJKliZsgkSVLtFEuI+egkSZKk6kRTZ1lWzpKlJElSxcyQSZKkWmqjBJkBmSRJqqk2isgsWUqSJFXMDJkkSaqhcJalJElS1ZxlKUmSpKYxQyZJkmonaKsx/QZkkiSpptooIrNkKUmSVDEzZJIkqZacZSlJklQxZ1lKkiSpacyQSZKkWmqjBJkBmSRJqqE2W/fCkqUkSVLFzJBJkqRacpalJElShQJnWUqSJKmJzJBJkqRaaqMEmQGZJEmqqTaKyCxZSpIkVcwMmSRJqiVnWUqSJFXMWZaSJElqGjNkkiSpltooQWZAJkmSaqqNIjJLlpIkSRUzQyZJkmoncJalJElStcJZlpIkSWoiM2SSJKmW2ihBZkAmSZJqqo0iMkuWkiRJFTNDJkmSaij6dZZlRDwKvATMAWZn5viIWAI4B1gFeBTYIzOnL8j1zZBJkqRaimjO1gsfyMz1M3N8uf814MrMHAdcWe4vEAMySZKkBbMLcFr5+jRgwoJeyIBMkiTVTjRx66EE/hwRt0TExLJtmcycWr5+ElhmQT+PY8gkSVI9NW8I2ZiIuLlh/+TMPLnTOe/NzCkRsTTwl4i4r/FgZmZE5IJ2wIBMkiQt7KY1jAubp8ycUv75dERcAGwEPBURYzNzakSMBZ5e0A5YspQkSbUUTfqv2/tELBIRi3W8BrYF7gIuBPYtT9sX+NOCfhYzZJIkqZb68VmWywAXRHHDQcCZmXl5RNwEnBsR+wOPAXss6A0MyCRJUi31VzyWmQ8D682j/Vlgq2bcw5KlJElSxcyQSZKk+un9oq4tzYBMkiTVVPtEZJYsJUmSKmaGTJIk1U5gyVKSJKlybRSPWbKUJEmqmhkySZJUS5YsJUmSKtaTxx7VhSVLSZKkipkhkyRJ9dQ+CTIDMkmSVE9tFI9ZspQkSaqaGTJJklQ74bMsJUmSqucsS0mSJDWNGTJJklRP7ZMgMyCTJEn11EbxmCVLSZKkqpkhkyRJteQsS0mSpEqFsywlSZLUPGbIJElS7QTtVbI0QyZJklQxAzJJkqSKWbKUJEm11E4lSwMySZJUS86ylCRJUtOYIZMkSfUTliwlSZIqFfgsS0mSJDWRGTJJklRPbZQiMyCTJEm15CxLSZIkNY0ZMkmSVEvOspQkSapYG8VjliwlSZKqZkAmSZLqKZq0dXebiO0j4v6ImBQRX2v658CSpSRJqqn+mGUZEQOBnwLbAI8DN0XEhZl5TzPvY4ZMkiRp/jYCJmXmw5n5GnA2sEuzb2KGTJIk1U7Qb7MslwcmN+w/Dmzc7JsYkKlbt916y7TFhg18rOp+LMTGANOq7oTUz/y+r87KVXegJ2699ZYrhg+OMU263LCIuLlh/+TMPLlJ1+4RAzJ1KzOXqroPC7OIuDkzx1fdD6k/+X2v7mTm9v10qynAig37K5RtTeUYMkmSpPm7CRgXEatGxBBgT+DCZt/EDJkkSdJ8ZObsiDgEuAIYCJyamXc3+z4GZFLr69dxDFKL8PteLSMzLwUu7ct7RGb25fUlSZLUDceQSZIkVcyATJIkqWIGZJKkWoiIwRGxZEQ4/lltx4BMakMR/bR+tdQPymcJAuxN8UzBxSvsjtQnDMikNtLwg2u7iHh3RAyttENSE2TmnPLlHsAs4PkKuyP1CQMyqY00/OC6ANgcmF1hd6SmiYgBwJnAS8CcKAzs5m1SbRiQSW0mIsYA1wE3NwRoUi2VgRgUq6N/E/ggsHwW5pTnDGw4T6olv4GlNtGQLViLIjO2ZcOxAf7AUh1l5r/Kl+OAV4ClgLsi4icR8d7ynDkN50m15MKwUpuJiJspHoQ7A/hv4NzMfK3h+EAzZ6qbcqLK+sCmwPuAtYERwGTgMuDSzPxndT2U3h4DMqmNRMQIYD/gXcAWwNLAJOBq4PzMvKG63km9FxGRnX5Qld/nG1J8j28ErE5R8VknM2f1eyelJjAgk9pQWb58F8XA/vcAawLDgGeA32bmSRV2T+qViBgMbEwRhN0CPJSZT5bHlqYozw/PzF9X10vp7TEgk9pMRIzMzBcb9hehyCK8F9gJODszfxwRAxx3o1bV8f0ZEasDRwPbAFOBdwKPAjtl5r0VdlFqKgMyqeYafnAtC3wY2AFYB7gJOC8zz2s4d3ng+cx8ZV6lIKlVdIx1jIjzKErvvwDOAr4LfIRi8sqSwCHAiZn5eGWdlZrAWVdS/XUEVccDXwFeBX4I7EaxsjkRsVxEDM/MKZn5CoDBmFpZGYwNpMiM/Twzzyy/Z3em+EVjJkVAthlFKVOqNQMyqcY6slwRMQ6YAHw6M3fPzBOB6cCfylM/AfyiHG8j1cU4ilmUCRAR76AYD3l2eXwIsArwWBWdk5rJgExqDzsC9wB3AkTEDsBQ4Iry+OO8uY6TVBePUvxisV+5P5Hie/y2chmMDwBk5h2V9E5qIgMyqcYayo6PAstSBF0Anwcuz8wnyhlq7wVeLMeO+e9etZCZrwInAJtHxOnAgcCvy+/7jwB7Ac6sVFvwf8xSe7ieYkmL/crB/e8HflkeW5ficTNnlPvR/92Teqb8BYKIWDUilsrM3wMfpRjEPwj4akT8D8X391XAcZV1VmoiZ1lKNRcRQzLztYjYAziVIuAaTjGmbEngc8Czmbl1hd2UeiUi/g4cm5kXlPvjKH65WIfi+/oS4EqfOqF2YUAm1Vg5juY/Kco408s1m/YBtgXGAw8AVwI/yMxHXHtMraqccLIRcB+wGMUCsO/MzIcr7ZjUTwzIpJopF3rNzJwREXsBP6WYeTatXCpgMWBRYCYwKjOdgaaWFxFbAt8GBlMEZHOA3TLzgU7nDQJWpihhXuzyLWoXBmRSzUTEbsB2wDXAF4CbM/PT8zl3EWBAZr7Ufz2Ueq8cO7Zzue0DPAvcRZEp+yvwj8x8ujz318BSmblTRd2Vmm5Q1R2Q1HNldmAYxXP9tqbIFEwuM2XXZ+b/NZw7GPge8Bfgwgq6K/VYZr4OnB8RtwPrAecBmwK7Ujzya1JE3AS8TlGSP7Sqvkp9wQyZVFMRcSrFD6unKQby/x/F45L+t/xzLMWaTe/KzLt8VJJaWTkeMsrHgA3JzNfK9rUpHgm2NbA8RSLhhszcs7reSs1nQCbVTMOzK7cEBgL3UpR5dgJWA2YBM4ClgIczcyuDMdVBGZR9HpgG/KF8PFLj8U2AfwF3ZOasCroo9RkDMqlGGh6VNAwYSbHY66sNxzekCM7WAe4A/pSZ/+x4UHM1vZa61vAg8Y8CXweOzMyzy2PLUfyicX1Z1pTakgGZVCMNP7gOpBjY/8vMvMQMmOqsIev7N+B24BvlMi67Al+iGEv2AnBgZp5TZV+lvuJK/VKNNGS5vgrcDdxQ7q8WEcdFxOURMQGKH3JV9FHqrTIYWxJ4J0WpcnqZBT6BYrblzhTjIieUE1uktuM3tlQTDVmE91Ks03RaZj4bESOAPwAjgCnAWRGxVWb+rcr+Sr20ODAZWD4iFgWOAoYCe2Tmq2Xp8lCKUv1z1XVT6hv+Bi3VR0dJ8j3AY8C0cmmLbwOjgF0oHrZ8J7BjJT2UFlBmTqLI+v6E4tmsW1OULl8ts2X/BszMTIMxtSUzZFJNNIwRuwr4LLAVxbP99ga+Wg7eHwxMBUbDm1m1Kvor9VZmfjwiDqD4peNnFBNTADagmEV8UlV9k/qag/qlmomI4cCPgU8CzwNHAj8pZ1++i2JV810z82oDMtVdRIyiWNh4ELB156UwpHZhQCbVVEQMAVbOzAfL/XHAdygeyDy+0s5JvRARQ4H3U6yd9zpwY8czWMtB/JsDL2fmTdX1UupbliylGoqIsZk5FXiwoXkcMAT4bnmOa4+pZTVMUlkL+AawJ8XTJgYCj0XEP4A/U6zKf3WFXZX6hRkyqcU1/OBakqJMOQGYTvHD63rg6sx8ouNcin/XBmJqaQ1r6p0FrADsRjGLcj+KsWMfoCjJ3wlcmZnfrayzUj8wIJNaXMPq/L+nyIJdCuxLkUl4FXgCuIZiTbKrMvOlqvoq9UZZjnwa+GRm/jEiHgJOyMzjI+JDwC8ovsd/lZlHVtlXqa8ZkEktrCEYW41iYczdM/PPEfEK8GWKNZm+BMymeMbfpzLzkup6LHWv4ft6e+B7FMu0LALcAmyZmTeX511KMaD/9MycUVmHpX7gOmRSa4vyz72BO8tgbE9gJnBGZn6PYuzNEOBcwLE2ankNS7i8Ctxavl4XeIY3v+ehKMlvYzCmhYGD+qUW1rBkxSjg7+XrbSlKlB0/uO6nWJvsd/7gUs1cC7xMUba8hSLjux1wU0RsTrHQ8cXVdU/qP2bIpBYXEQOByykWfIXi2X4rUSwPAMW/443KNiIiOl9DajUdg/rL8uQcirGQvwYOi4hnKL7nX6d4nqXU9hxDJtVERAzKzNkRsRPFuJrfUWTKPgKsn5lLV9k/qTci4j8oSu0XN05EiYj3AZuVu2dn5qMVdE/qdwZkUouLiFWA5zPz+Ya2vYCvAWsBNwI/KGepufaYWlantccuBI7PzJ+Ux4YA6wG3ZebsKvspVcGSpdSCyjIlEfF+itX3t2s8nplnZeZ6wJLAhzLzj2W7wZhaWcfPnM8Bj1EEZUTEesBpFI/9ejIiPlVN96TqGJBJre3rwAjKmWgRsWxE7B8RZ0fErsBLmflcpT2Ueqgh87Ud8HuKxY0BfgSsCXyVIijbIyIW6f8eStVxlqXUgsoVzBcH3gPsBEwqD50CjKcY4H8qxfpjF1XSSWkBRMQYiuzY8HItsoOB91GMg7y7XBz2R8CqwF0VdlXqVwZkUovpWDQT2JJiOYCHy/Z9gO2BnSmWuvg18KGIuDwzX5/f9aRWkpnTIuIm4PsRsTOwPPCTMhgbBCwBLJ6ZBmNaqBiQSS2mYdHMu4DkzWf6/TdwYmZeDlD+UFvXYEytLCJGUQRYj3a0ZeaXI+JBYBuKrG/H0yXeCXwU1x7TQshZllKL6DxDMiIGUwzo/xwwDPgx8P3MfDIiVgD+DPwmM491dqVaVUT8FnglMz8dEUsDQ4EpDYseNz5K6ZfApsAemXl3RV2WKmFAJrWIiPgC8OfMvKtjeYCyfVlgaeCBzHw1IkYA3wb2ANbLzOcbypxSS4mID1Akfq+JiCsoVuM/g+LZrI9k5rTyvKHABymCtysq67BUEQMyqQVExKIUK5XvkJnXR8RngXuAfwLPNAZbETGeYg2yGzPz+43Bm9RKImJ0p/Xz9gMmAhsDzwFXAJdRzCJ+KDNnVdJRqQUYkEkV65zdiog1KIKxl4C/UazVdBMwqcyGDQFGUyx5MdPsmFpVRJxDser+T4DLM/POsn0p4ABgf+AdFBNXLgOuBC70FwwtjAzIpBYQ8f/t3XusZWV5x/HvbwYYEIEWS5EgKmlFuYhAlIuUAZES0LSGAYuBGogoFwWq2KSxTVtKbSxeYjSmsRZrJTLcRCnYyk1KGYwK1MIAU5QKdILYAbkKzDACT/943y2744wMcM5Z+8x8P//svddea6/nnLCZ57zrWc+Tjapq5djrecC7gROBPWh3W14B/CtwI3BPVa0YIlZpbSU5CDgGOJR2qfKbwD/TkrN7+z47AycAJwPfqap9BwpXGpQJmTSwJK8C3gdcBNwBPDl+52SSbWn/YB1DaxHwKHBSVZ0/QLjS89aL+Y8BzuybHqSthp0PXF5Vj/f9tqmqnwwTpTQsEzJpYH3I8oXA47TVr4XAdcCPaQXO43de7gF8BDizqm60fkyTbHT3b5KTgf1p/43fQBv5dRhtduWttJWzb1TVosGClQZmQiZNgL6CcCDtH6l3AnfTkrKLaUnaT4Hl1oppNkqyDPgYbZh49W2/RqshO5M2xu9rVXXEcFFKw7IxrDQBquq+JBcCT9NqxlYCewNH0+b9fRW4NMl/A8vsOabZIsnutD56S8b/oOh3X34qya7A9bTh4tJ6y+Hi0sB6A1iAU2jtLP4M2I82x3If4Brgw8BlwL8Dm818lNIL9gCwDHhvH420qluABVX12MyGJU0WV8ik4T3VH98DXA18fayo//4kP6SNUHoDsNBGsJpNqmppko/Tmhl/KsmXaXcJ39fvsPwDwK78Wu+ZkEkD6yNjXkIr4n/1qrMpewJ2Dq0Q2jsrNRudB7yCttJ7LHBTkpW0HmV3Ap8cLjRpMnjJUpoAVfUErYD/4CSnJdk6ydyxXTYF5gOP9P1dHdOsUVWPVdXptBFgf0S7jPkAcDpwpHMrJe+ylCZGXyX7LK2Q/ypah/6HgNfTLuvcXVWHOkhc6wJbtkj/nwmZNGGSvIN2aWcf4GFajdk3ab3HfmBCJknrHhMyaUKsZqblxsCetJWxpcNFJkmabiZk0oRJEtp308s5krSesKhfmjDV/FIy1hM1kvi9laR1jP9jl2aJscuZ1yTZYdBgJElTyoRMmgVGLTCSHEkr9n9g2IgkSVPJGjJpFhjdWZnkWuD2qjp+6JgkSVPHFTJpYKs0gF2tnoxtQZtvuXD6o5IkzSQTMmlgz9VTbCxhOw64G7h+umOSJM0sZ1lKM2jUnTzJy4C3A+8C7qGNTfpeVa2uNmxUV3ACbbj4EzMTrSRpplhDJg0gyZeBtwJfBU7tm5cBF9AGiC+uqsfG9t8euAPYvapumeFwJUnTzIRMmiFjq2MHAucCC4AfAUuBDwC7ASfRVsQeBParqtv7sXsDhwB/7dgkSVr3eMlSmjnpj8cC11XVt5OcRkvKzqOtjP2ctnJ26SgZA6iq7ya53u79krRusqhfmiFjK1tbAf/Zny8ArgBWVNWjtMTsXuCc8WP7nEuTMUlaR7lCJs2gJPNoydbou/cIsFlV/Xzs9c6j90cDx8vaAklap5mQSTOoqp5MciEwr2+6DviTJItp3fcPAZ6qqpv7/iZikrQeMCGTZlhVPQk82V+eA+wCnAZsAtwHfASe7c4/SJCSpBnlXZbSBOh3Ub4cuBZ4yJUxSVq/mJBJAxq1whg6DknSsLzLUhrQKBlLMmqJQZI5/XH7JNsNFZskaeaYkEkTYA2XKBcC82c6FknSzDMhkybIqN9YktcAewGOSZKk9YAJmTRNRpchk8xLsrZ3NI++kx8AvgcsmY7YJEmTxYRMmn6nAQetzY5jbS6OBr5SVU9NW1SSpIlhQiZNsSQbJtlurC7sb4ANxwv313DcqJh/PvBS4JLpjVSSNClsDCtNvZ2BDyZZAcwF/hdYNF64P2p3kWRjYFvgTtofSM8AH6TNt7xnxiOXJA3CFTJp6j0MLAf2Bd4NFPDnSX4/ydbwbLsL4CjgY31c5ejy5O8A59ocVpLWHzaGlaZJktcC1wM3AdvQVr9+RCvWvxq4H/gSsLiqTuyXLDcFDgCurKoVQ8QtSZp5JmTS8swZlwAABa9JREFUFOu1YnNoA8QPrqqLk+wILADeCrwKWAFs2R/3r6qldu2XpPWXCZk0zZJsMH63ZJL9gbfQBoxfWVU39v5jfhklaT1lQiZNoVFi1S8/vgk4HHg9Lfk6r6rOGzRASdJEsqhfmlpz++P7gLOB44CNgK2Bs5MsTfKHoxYXkiSBK2TStEjyE+A84E+ranmSVwK7Ah8GXgEcXlWLh4xRkjQ5/CtdmiJjo5L2ATYEvlBVywGqamlVfQN4F7AZcORggUqSJo4JmTRFxoryNwGeAvYESLJRko36PsuAi4D9BwlSkjSRTMikqfdvwA+B45K8vKpWVtVKgCSb09pe3N1fz13jp0iS1hsmZNIU6r3ECjid1gz2f5JckGRBkrcBFwC7A5/th1jEKUmyqF+aLr2Q/1jgYGCPvvkW4DNVtXCouCRJk8eETHqRxnqPbQQcCpwCPEobEL4YeLy/DnB/Vf1ssGAlSRNpg6EDkNYBc2lF/McDHwXupF2K/CTwNG1u5SX90dFIkqRf4gqZNEWS3EHrPfa5qlqW5DeA3wOOAfaiJW4fraozBgxTkjSBLOqXpkCSebQVsBt6Mjanqn5aVV+qqgOAHYFPALf2/b27UpL0C66QSS9CkrlV9XSSQ4D3Az+uqpPG3t8AeKaqvFQpSVojEzJpCiS5EtgJ2Bz4PPDFqrp97P05ACZmkqTVMSGTpkCSVwO/CxwBvBZYCSwBLgUurar7BgtOkjTxTMikKdQvUe4AHA4cCGzb37qqqt4/WGCSpIlmQiZNkySb0BrCngDcWlUfH9WcDRyaJGnCmJBJM2Cs+D/ll06StAoTMkmSpIHZh0ySJGlgJmSSJEkDMyGTJEkamAmZpImR5OkkNyW5NcmFSV7yIj7rn5Ic0Z+flWSnX7HvAUne/ALOcXefWbpW21fZ57Hnea7Tk/zx841R0uxgQiZpkiyvqt2qahdac90Tx9/sfd6et6p6b1Ut+RW7HAA874RMkqaKCZmkSbUI+O2+erUoySXAkiRzk3wiyQ1JFic5ASDN55L8IMlVwG+OPijJNUne2J8fkuT7SW5O8q0+ZeFE4EN9dW6/JFsluaif44Yk+/ZjX5bkiiS3JTkLyHP9EEkuTvIf/ZjjV3nv0337t5Js1bf9VpLL+jGLkrxuKn6ZkibbC/prU5KmU18JOxS4rG/aA9ilqu7qSc0jVfWmJPOAbye5AtidNrZqJ2Br2uiqf1zlc7cC/gGY3z9ry6p6MMnngceq6pN9v4XAp6vquiSvBC4HdgT+Eriuqs5I8nbguLX4cd7Tz7EJcEOSi6rqAWBT4Maq+lCSv+iffTLwBeDEqrojyV7A39GmPkhah5mQSZokmyS5qT9fBHyRdinx+qq6q28/GNh1VB8GbAG8BpgPnNsnIdyb5OrVfP7ewLWjz6qqB9cQx0HATskvFsA2T/LSfo4F/dh/SfLQWvxMpyY5rD/frsf6APAMcH7f/hXga/0cbwYuHDv3vLU4h6RZzoRM0iRZXlW7jW/oicnj45uAU6rq8lX2e9sUxjEH2LuqVqwmlrWW5ABacrdPVT2R5Bpg4zXsXv28D6/6O5C07rOGTNJsczlwUpINAZLskGRT4FrgyF5jtg3wltUc+11gfpLt+7Fb9u0/AzYb2+8K4JTRiySjBOla4Ki+7VDg158j1i2Ah3oy9jraCt3IHGC0yncU7VLoo8BdSd7Zz5Ekb3iOc0haB5iQSZptzqLVh30/ya3A39NW+78O3NHfOxv4zqoHVtX9wPG0y4M38+wlw0uBw0ZF/cCpwBv7TQNLePZuz7+iJXS30S5dLn2OWC8DNkjyX8Df0hLCkceBPfvPcCBwRt9+NHBcj+824B1r8TuRNMs5y1KSJGlgrpBJkiQNzIRMkiRpYCZkkiRJAzMhkyRJGpgJmSRJ0sBMyCRJkgZmQiZJkjQwEzJJkqSB/R8z0JgbLTH4PQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 1080x504 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YKLMrLcfBt50"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}